Maxwell's Demon and Data Compression 
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In an asymmetric Szilard engine model of Maxwell's demon, we show the equivalence between 
information theoretical and thermodynamic entropies when the demon erases information optimally. 
The work gain by the engine can be exactly canceled out by the work necessary to reset demon's 
memory after optimal data compression a la Shannon before the erasure. 
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I. INTRODUCTION 

Entropy is one of the most cardinal concepts in the 
modern science. The idea of entropy plays a crucial role 
in not only thermodynamics, but also the physics of black 
holes and information science, including quantum infor- 
mation, to name a few. The interplay of entropy in clas- 
sical physics and information science has been studied 
intensively since it was first pointed out by Brillouin in 
the general context [ll,[2|- Then, this idea was clarified b" 
Landauer in the form of information erasure principle 
Landauer's work opened up a way to relate the informa- 
tion theoretic and thermodynamic entropies. In order to 
obtain further insights into the relation, we need a simple 
and specific model. In this sense, the most well-studied 
is Szilard's engine and Maxwell's demon. 

Since Maxwell mentioned an apparent violation of the 
second law of thermodynamics by a fictitious intelligent 
being in his textbook in 1871 this paradoxical prob- 
lem has been debated intensively under the name of 
Maxwell's demon Q. Towards its solution, Szilard 
devised in 1929 a one-molecule engine model, which inge- 
niously distilled the essence of the problem and made him 
realize the significance of information in the thermody- 
namic process Q- Although it still took some time after 
Szilard, a satisfactory solution that lets the demon down 
was eventuallyreached, based on the idea of Landauer \^ 
and Bennett Q . The overall consensus we share today is 
that erasing information in demon's memory causes an 
entropy increase, which, with demon's best effort, pre- 
cisely cancels out the work gain when closing the ther- 
modynamic cycle. 

The physical process of information erasure has been 
investigated from various aspects: noteworthy examples 
are two derivations of the entropy increase by Shizume @ 
and Piechocinska [l^ . They both showed that the lower 
bound of the entropy increase for erasing one bit of in- 
formation should be fc^ ln2, where ks is the Boltzmann 
constant. This entropy increase is exactly the minimum 



amount to circumvent the contradiction with the second 
law in the demonic paradox. 

This specific example suggests a possible way to link 
a certain entropy like quantity with the information en- 
tropy. This could be achieved by considering an oper- 
ational model to carry out information erasure with a 
dynamical process intrinsic to the system of interest. 

In the present paper, on the basis of the Shannon com- 
pression of demon's memory before erasure in the asym- 
metric Szilard engine model, we prove that the optimal 
cost of information erasure is 

fcsln2-i/(p), (1) 

where H{p) is the Shannon information entropy 



H{p) = -_plog2p- (l-p) log2(l -p) 



(2) 
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with p being the ratio of the proportional division of the 
cylinder by the partition. Therefore, the entropy decrease 
of the Szilard engine exactly cancels out the entropy in- 
crease by the optimal information erasure of the demon. 

This paper is organized as follows. In Sec. HIl we reca- 
pitulate the resolution of the Maxwell's demon paradox 
by Landauer and Bennett in the standard symmetric Szi- 
lard engine model. In Sec. IIIIl we propose the protocol 
of the demon in the case of an asymmetric Szilard en- 
gine. We show that the erasure work can be minimized 
by Shannon's data compression. In Sec. IIVI we consider 
a different scenario, where heat baths of different temper- 
atures are used for the engine-demon system. Section Ivl 
is devoted to summary and discussions. 



II. SYMMETRIC SZILARD ENGINE AND 
ERASURE OF MEMORY 

In this section we briefly review the Landauer prin- 
ciple of information erasure in the standard symmetric 
Szilard engine model ,3]. The Szilard engine consists of 
a one-dimensional cylinder, whose volume is Vq, contain- 
ing a single-molecule gas and a partition that works as a 
movable piston. 

The operator, i.e., a demon, of the engine inserts the 
partition into the cylinder, measures the position of the 
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molecule, and connects to the partition a string with 
a weight at its end. These actions by the demon are 
optimally performed without energy consumption Q. 
Throughout this paper, the demon's memory is also mod- 
eled as a single-molecule gas in a box with a partition in 
the middle. Binary information, and 1, is represented 
by the position of the molecule in the box, the left and 
the right, respectively. This model of symmetric memory 
has an advantage that reading, encoding, and computing 
over bits require no energy, making it consistent with the 
scenario of reversible computation. 

The following is the protocol to extract work from 
the engine by information processing of the demon (see 
Fig. [T]), where we denote "SzE" for the Szilard engine 
and "DM" for the demon's memory at each step of the 
protocol. Initially, the molecule in the cylinder moves 
freely over the volume Vq. 

Step 1 (SzE) The partition is inserted at the center of 
the cylinder. 

Step 2 (SzE, DM) The demon measures the location 
of the molecule, either the left ("L") or the right 
( "R" ) side of the partition. The demon records the 
measurement outcome in his memory. When it is 
L (R), his memory is recorded as "0" ("1"). 

Step 3 (SzE) Depending on the measurement outcome, 
the demon arranges the device differently. That is, 
when the molecule was found on the left (right) 
hand side, i.e., the record is (1), he attaches 
the string to the partition from the left (right). 
In either case, by putting the cylinder in contact 
with the heat bath of temperature T, the molecule 
pushes the partition, thus exerting work on the 
weight, until the partition reaches the end of the 
cylinder. The amount of work extracted by the en- 
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W = kBT\n2, 



(3) 



as can be seen by applying the combined gas law 
in one dimension. 

Step 4 (SzE) The demon removes the partition of the 
engine, letting the molecule return to its initial 
state. 

Step 5 (DM) The demon removes the partition of his 
memory to erase information. 

Step 6 (DM) In order to reset the memory to its initial 

state, the demon compresses the volume of the gas 
by half. 

In order to complete the cycle for both the Szilard en- 
gine and the memory, the demon has to reset the memory, 
which follows the erasure of one-bit information. Fol- 
lowing is a more precise explanation about the physical 
process of information erasure and memory resetting de- 
scribed in Steps 5 and 6. The box is in contact with the 



Szilard Engine 

Vo 



step 1 O 
Vo/2 ,Vo/2 




St ep 2 O step 2 



Demon Memory 



R Record 



Step 3 JJ^ 



Step 4 O 



Step 2 



Step 5 



^TT^ Step 6 



FIG. 1: (Color online). A protocol of symmetric Szilard en- 
gine (black/left side) and demon's memory (red/right side). 
This figure shows an example in which the molecule was found 
in the right hand side of the cylinder. In demon's memory, 
the state after removing the partition is denoted by "*" . 



thermal bath at the same temperature T as that of the 
engine. The record in the memory can be erased sim- 
ply by removing the partition, since the location of the 
molecule becomes completely uncertain. To bring the 
memory back to its initial state, e.g., 0, one has to com- 
press the gas by half by sliding a piston from the right 
end to the middle. The necessary work for this compres- 
sion is fcBTln2, which exactly cancels out the work gain 
by the engine (jSj. Here, we have taken the result by 
Piechocinska for granted that the erasure of a single bit 
of information requires a work of at least fcBTln2 10]. 

Let us look at the same process in terms of thermody- 
namic entropy. By Steps 1 and 2, the volume of the gas 
in engine is halved, regardless of the measurement out- 
come. As the entropy change of an ideal gas under the 
isothermal process is given by AS := S{V') — S{V) — 
fc_B ln(V^'/F), the entropy of the engine is lowered by 
fcBln2. The isothermal expansion in Step 3 increases 
the entropy of the gas by fc^ ln2, while that of the heat 
bath is decreased by the same amount. As far as the 
Szilard engine and its heat bath are concerned, the net 
result is an entropy decrease of /cs In 2. Nevertheless, this 
is exactly canceled out by the entropy increase due to in- 
formation erasure and reset performed in Steps 5 and 6. 

These last two steps are of crucial importance when 
closing a cycle of the memory. Information erasure in 
Step 5 is an irreversible process and increases thermody- 
namic entropy by fcBln2. The isothermal compression 
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to reset the memory in Step 6 requires work and dis- 
sipates entropy of feg In 2 to its heat bath. This is the 
essence of Landauer-Bennett mechanism that resolves the 
MaxweU's demon paradox. 

Now let us slightly generalize the Szilard engine model 
to an asymmetric one in such a way that the partition 
is inserted to divide the whole volume Vq into pVa and 
(1 - p)Vo with < p < 1 (See Fig. ^. A straightfor- 
ward calculation shows that the work extracted by the 
asymmetric Szilard engine is 



(4) 



where S'(p) = —p\np—{l—p)\n{l—p) If the memory 
is reset after every cycle of the engine, the amount of work 
consumption is 



AW = kBT\n2- keTSip) > 0. 



(5) 



However, one may wonder if the gap, AW, could be 
smaller by employing a better strategy. In the follow- 
ing section, we show an information theoretical protocol 
that fills the gap optimally. 
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Demon Memory 
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FIG. 2: (Color online). The model of an asymmetric Szilard 
engine. The position of the partition in demon's memory is 
the same as that in the case of the symmetric version. 



III. ASYMMETRIC SZILARD ENGINE AND 
ERASURE OF COMPRESSED MEMORY 

We are going to show a protocol in which the demon is 
clever enough to reduce the work for the erasure by us- 
ing the Shannon data compression [12j in the asymmetric 
Szilard engine introduced in the previous section. First, 
the demon accumulates the data of N cycles, which we 
assume is very large. The data contains uneven number 
of O's and I's corresponding to the measured position of 
the molecule in the engine. The relative frequency of 
O's is obviously p, while that of I's is 1 — p. According 
to Shannon's noiseless coding theorem, the demon can 
compress the data to a shorter one, whose length will be 
Ng :— NH{p) < N ai shortest. Coding does not cost 
any work if we employ reversible computation [Tsj l , pro- 
vided that the memory is symmetric as remarked before. 
If asymmetric memory were used, even the NOT gate, 
and therefore generic computation, cost energy, which 
makes our task less transparent. See, e.g., Refs. EE]- 
Then, he erases the shortened data string with the work 
kBT\n2 ■ Ns = ksTNSip). Therefore, the difference 



between the work to reset the memory and the work ex- 
tracted by the engine approaches zero, 

AW{optimal) = keTN^ - keTSip) = 0, (6) 

for a very large N . 

To be more precise, we write down the optimal protocol 
below. 

Step 1 (SzE) The partition is inserted to divide the vol- 
ume into two parts, pV^ and (1— p)Vb, in the initial 
configuration of the cylinder and a single molecule 
is either on the left or the right of the partition. 

Step 2 (SzE, DM) The demon measures the location 
of the molecule and records either for the left (L) 
or 1 for the right (R) and keep the result in his 
memory. 

Step 3 (SzE) Depending on the recorded information, 
the demon arranges the device differently. That is, 
when the molecule was found on the left (right) 
hand side, i.e., the record is (1), he attaches 
the end of the string to the partition from the left 
(right). In either case, the molecule pushes the par- 
tition which is now movable to the very end of the 
cylinder. 

Step 4 (SzE) In order to go back to the initial config- 
uration, the demon disconnects the cylinder from 
the attached device. 

Step 5 (DM) The demon repeats Steps from 1 to 4 for 
N times, keeping the A^-bit string in his memory. 
Then, he compresses the A^-bit string to the mini- 
mum length NH{p), according to Shannon's noise- 
less coding theorem. We break up the A" = mn bit 
string into m blocks of n bits. 

A brief description of the data compression is the 
following. 

First, punctuate the Af-bit string by n bits so 
that we have 2" sequences with relative frequen- 
cies p", (1 — p)p^~^, . . . , (1 — p)" so that we can 
encode the sequences into strings of bit length of 
-log2P",-log2(l -p)p"-\...,-log2(l -p)", if 
they were integers. Roughly speaking, the average 
bit length would be 

I — n V / 



= nH{p). 



(7) 



The right hand side of Eq. ([7]) coincides with the 
shortest average bit length shown by Shannon [l^ . 

Step 6 (DM) The demon repeats the process of n turns 
m times so that the total amount of bits to be 
erased is 



SKmnH{p) = NH{p). 



(8) 
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FIG. 3: The protocol of data compression for demon's memory. The dashed blocks express the trivial initial memory state "0" 
after data compression. 



The change in thermodynamic entropy is calculated 
in the same manner as in the above case of symmetric 
engine. The volume of the gas after Step 2 becomes pVb 
with probability p or (1 — p)Va with probability 1 — p. 
The entropy of the gas after Step 2 is thus decreased by 
— plnp— (1— p) ln(l— p), which is equal to S{p) in Eq. (g]) 
and is to be canceled out by the later steps of information 
erasure and memory resetting. 

Let us treat Steps 5 and 6 more precisely. According 
to the Shannon source coding theorem for the symbol 
codes [ij], for 7i-bit string, there always exists an opti- 
mal code such that the averaged code length S satisfies 
nH{p) < S < nH{p) + 1. The well-known example of op- 
timal codes is the Huffman code for the encoding proce- 
dure, see Ref. [IB]. It is reminded that the demon breaks 
up the A''-bit string into n-bit block, i.e., N = nm -\- 5, 
where < (5 < n. The extra (5-bit string cannot be 
encoded. When the demon optimally encodes an TV-bit 
string, its length is longer than NH{p) bits hy m~v5 bits 
at worst. As the discrepancy is bounded as 



string over N bits is 



+ 5 = N - (n- l)m > N - 



{n - 1)2 + nri^ 



(9) 



it can be minimized when n — 1 — m. Thus, we obtain 
n = m — 0{\/N). The extra bits are at worst 6 ~ 
0{-\/N). Therefore, we can more precisely express the 
average length of the optimally compressed data string 
as 



S = mnH{p) + 6^ NH{p) + OiVN). (10) 
The averaged work necessary to erase information in this 



fcBTln2 • S 
N 

^kBT\n2-H{p) + 0{^] . (11) 



W (erasure) 



On the other hand, in Step 3, the amount of average 
work extracted by the engine over N cycles is given by 



W{engine) = ksTSip) + O ( 



(12) 



where Siji) = ln2 • H{p) as can be seen by applying 
the combined gas law. It is reminded that S{p) is the 
thermodynamic entropy as discussed before. 

It is now clear that the work for information erasure 
of demons's memory (jlip and the work from the asym- 
metric Szilard engine ([T^ agree for sufficiently large N . 
The above argument leads us to conclude that the infor- 
mation theoretical entropy is equivalent to the thermo- 
dynamic entropy when optimal information processing is 
physically executed. 

Note that, for the symmetric Szilard engine, we do 
not need to compress data because the number of O's 
and I's are equal. Also, the erasure model for non- 
equiprobability distribution of the memory was consid- 
ered in a simple thermodynamic process [B, [iTj . The 
amount of work for this process coincides with the opti- 
mal one (jlip. 
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IV. ANOTHER HEAT BATH AT A LOWER 
TEMPERATURE 

The reader might question why the demon resets his 
memory at the same temperature (say, Th) as the heat 
bath for the Szilard engine and wonder what if the era- 
sure is executed at a lower temperature, Tl{< Th), be- 
cause the compression of the memory space would then 
require less work. With two heat baths of different 
temperatures, some nonzero work W can indeed be ex- 
tracted, however, the amount of entropy increase is al- 
ways larger than or equal to fc^ ln2. That is, in terms of 
entropy balance there is no difference from the case with 
a single heat bath. Hence, the demon's attempt to outdo 
the second law ends up in vain, as we naturally expect. 
An example of erasing process with two heat baths that 
achieves the bound is depicted in Fig. 2] and explained 
in its caption. 

When the optimality is achieved, the entire compound 
system, the engine and the memory, simply works as a 
single engine; it converts a part of heat absorbed at Th 
into the work W and throws the residual energy away 
to the heat bath at T^. The overall thermal efficiency 
is equal to 77 = W/Qh = 1 — Tl/Th, where Qh is the 
amount of heat flowed from the heat bath at Th to the 
Szilard engine, thus effectively the same as the Carnot 
engine. 

Let us make a remark to avoid a potential confu- 
sion. Despite being equivalent to the Carnot engine, the 
engine-memory system does not work reversibly in the 
context of information erasure. While the system can be 
run in the reverse direction, the erased information can 
never be restored reliably. 



V. SUMMARY AND DISCUSSIONS 

We have shown in the asymmetric Szilard engine that 
the work extracted by Maxwell's demon is asymptotically 
canceled out after a large number of cycles by the work 
to reset the memory after optimal data compression. We 
have described an explicit protocol and shown its opti- 
mality by making use of Shannon's noiseless coding the- 
orem. The key point is data compression before infor- 
mation erasure of the memory and this argument makes 
the seminal work by Landauer and Bennett more gen- 
eral and precise. The coincidence between information 
and thermodynamic entropies is now very clear, thanks 
to the demon's cleverest strategy. As a slight general- 
ization we have also considered the case of information 
erasure at lower temperature to see that the efficiency of 
the whole system can be only as efficient as the Carnot 
cycle and that there is no net gain for the demon. 

We would like to stress that the thermo-informational 
cycle has to be completed to correctly address the appar- 
ent violation of the second law in the context of Maxwell's 
demon. This means that no residual information should 
be left outside the engine-demon system after the cycle 
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FIG. 4: (Color online). The p-V diagram of an alternative 
erasing process with a heat bath of lower temperature. The 
erasure is realized with any path from the "*" state to the "0" 
state, which are denoted by a diamond o and a circle o, respec- 
tively. The solid black line represents the standard erasure by 
an isothermal compression (at temperature Th)- Although 
the demon may want to use a colder heat bath of tempera- 
ture Tl, the entropy increase due to the erasure cannot be 
smaller than fcBln2. The path, consisting of two adiabatic 
processes (red dashed lines) and one isothermal compression 
(blue dot-dashed line), in the figure is the optimal one in 
terms of entropy increase and attains the Landauer limit of 
ks In 2. Naturally, the entropy cost is the same even if the 
temperature of the memory is always Tl, while the Szilard 
engine is operated at Th- 



[18j . What makes the argument of demon important and 
interesting is this physical loss of information, otherwise 
it is merely a sequence of normal measurements. 

As briefly remarked in the introduction, the present 
work in the specific model suggests a general method to 
relate information and physical entropies by considering 
an operational process to erase information in the phys- 
ical system. One such example is the original derivation 
of the black hole entropy by Beckenstein [l^. He con- 
sidered a gedanken experiment of dropping a "particle 
of one bit" for information erasure which increases the 
area of the event horizon as a back action. He identified 
the amount of information loss with the change of the in- 
trinsic entropy of black hole. Also, there is a well-known 
derivation of the Boltzmann distribution on the basis of 
the principle of the maximum Shannon entropy under the 
energy constraint (20| . However, the physical meaning of 
the Shannon entropy there is not clear, though the opti- 
mal value coincides with the thermodynamic entropy. It 
would be nice if we could clarify the meaning of the max- 
imization of the Shannon entropy in terms of the optimal 
memory reset. 

The optimal information erasure would help us fully 
understand physical entropy in terms of information en- 
tropy, as Brillouin envisioned 
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